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ABSTRACT 

The Knowledge Engineering for Young Scholars (KEYS) 
Program was a National Science Foundation (NSF) program conducted at 
Louisiana State University during 1989 and 1990. The program's goals 
were to increase 8th~12th grade students' exposure to science > 
acquaint them with university research, stimulate interest in 
science, and build their confidence to pursue -science careers. To 
achieve this goal students were introduced to the concepts of 
artificial intelligence, expert systems, and knowledge-based systems. 
This document presents the evaluation report of the KEYS Program in 
six sections. The first section identifies the goal of the program. 
The second section provides information about the purpose of the 
program, a program description, student recruitment and selection 
criteria, program activities , career awareness activities , philosophy 
and ethics of science activities, follow-up activities, the project 
setting, a staff profile, and cost considerations. The third section 
describes the evaluation study of the program that included NSF pre- 
and post-participation survey forms, and interviews with KEYS program 
managers (n=2) , faculty mentors (n-2) , program teachers (n=2) , 
counsel ors (n*=2) , and random samples of student participants (n=10) , 
parents, of KEYS student participants (n=6) , and applicants who did 
not participate (n^lO) . The fourth section provides brief summaries 
of each groups' responses to specific interview questions, and 
results of the NSF surveys. The fifth section concludes that: (1) 
since the participants were already highly motivated and interested 
in science careers, little change in reported interest was evidenced; 
(2) student participants' self report on their confidence level for 
their first choice of a major in college was not reliable; and (3) 
students did report that they did work with university faculty, 
develop expert systems, and learned about careers in science. The 
sixth section presents recommendations concerning program length, 
effective activities, participant recruitment, participation fees, 
and program assessment techniques. An appendix provides sample 
abstracts of KEYS students' expert systems. (MDH) 
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I. Summary 



Since 1989, the National Science Foundation (NSF) has funded a Young Scholars 
Program which has as its goals to increase 8-12th grade students' exposure to science, 
acquaint them with university research, and stimulate interest in and build their confidence 
to pursue science careers. This is the evaluation of one representative program of this 
nation-wide NSF sponsored educational effort to recruit U.S. students into science and 
engineering careers. 

An NSF program, Knowledge Engineering for Young Scholars (KEYS), was 
conducted at Louisiana State University in Baton Rouge during 1989 and 1990. Student 
participants learned artificial intelligence programming techniques by developing expert 
systems. To accomplish this, high school students received specialized training and then 
worked one-on-one with faculty mentors in agriculture to develop a scientific decision- 
making model. 

The NSF Young Scholars Program conducted at LSU enabled talented high school 
students to experience college life and build confidence in their own ability to successfully 
complete difficult tasks. The KEYS program utilized the concepts of expert system 
programming as a vehicle for enhancing students ability to make career decisions. 

II. Background Information 

A. Purpose of the Program 

The LSU KEYS Program was designed to expose high ability high school students 
to research and career options in science and engineering through introducing the students 
to the concepts of artificial intelligence, expert systems, and knowledge-based systems. This 
was done with specialized class room training, one-on-one work with faculty mentors, field 
trips, the use of LSU's state-of-the-art computer laboratory, hands-on practice in decision 
making, training in personal and professional ethics, introduction to careers in science and 
engineering, and residential college life experiences. 

B. Description of the Program 

This program explored the decision making strategies that are used by experts in 
science and engineering disciplines. The participants were taught the fundamentals of 
knowledge engineering and expert systems (ES)/artificial intelligence (AI) programming and 
then they developed an expert system designed to model the decision making rules used by 
discipline experts. An eight-week program was conducted in 1989 and two four-week 
programs were held in 1990. During the first half of each program students learned the 
fundamentals of knowledge engineering, participated in careers in agricultural and life 
sciences seminars, were presented ethical issues in science and engineering, trained in 
technical and professional communication, and studied research methodology. During the 
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second half of the program, the students worked with a faculty mentor to develop an expert 
system in engineering, agriculture or life sciences. The students worked in the LSU 
Knowledge Based System Development Laboratory (KBSDL) utilizing state-of-the-art ES/AI 
work stations. 

The participants chose projects from a wide array of research topics in engineering, 
agriculture, and life sciences. The expert system included diagnostic tools, identification 
guides, trouble shooting programs, and simulation model interpreters. The scope of the 
work allowed the students to evaluate their interest in a field of science. By working with 
knowledge based system technology, the students learned the fundamentals of decision 
making and were given tools with which to make decisions relative to their own career goals. 

C. Student Recruitment and Participant Selection 

Fifty-nine outstanding high school students from Louisiana and Texas participated in 
the Knowledge Engineering for Young Scholars program during 1989 and 1990. Over 200 
students applied for the program. All students met the high school rank requirements and 
many excellent students were unable to participate due to the need to limit enrollment to 
20 participants per session. 

Efforts were concentrated on attracting students from smaller rural schools within the 
State of Louisiana. The program was advertised through brochures prepared in cooperation 
with the Division of Continuing Education at Louisiana State University. Program brochures 
were sent to the superintendents of public schools in all 64 Louisiana parishes, the 
superintendents of Catholic diocesan school boards (non-public schools), and science 
teachers and guidance counselors in all public and private high schools in Louisiana. The 
4-H agents in the Cooperative Extension Service were encouraged io nominate students. 
Attempts were made to identify promising minority students through the activities of liaison 
personnel, and other LSU faculty involved with science and engineering fairs. 

Each applicant was asked to provide: a completed program application to include 
information on gender, minority status, etc.; an official high school transcript of records to 
include GPA and rank in class; verification that the student was in the upper 20% of his/her 
class; a personal essay (250 words) describing the student's own interests, activities and 
experiences in science as it relates to food, agriculture and/or natural resources; and three 
letters of recommendation from a science teacher, a guidance counselor and other teacher 
or person to describe the student's current science-based abilities and activities, and his/her 
potential to benefit from the program and experience. 

Program participants were incoming 11th and 12th grade students of high academic 
ability or potential who had expressed interest in pursuing a career in science and 
engineering as it relates to food, agriculture, and natural resources. The 59 students who 
participated in the program included 32 females, 14 African Americans, 1 Pacific Islander, 
and 5 Asian Americans, Students from the entire State of Louisiana were encouraged to 



apply and as noted earlier, there was a special effort to recruit in smaller rural high schools 
with limited opportunities for high ability students. The average GPA of the entire group 
was 3.9/4.0. All the student participants who have graduated from high school have enrolled 
in college. 

D. Program Activities 

In 1989, there were twenty student participants and an eight week format was used. 
During the first 4 weeks, students were taught expert systems programming theory and 
techniques. During this time period faculty mentors made presentations describing potential 
projects and career opportunities in their particular discipline areas. Several tours of 
research labs were also taken. The lab sessions in the afternoon focused on use of the 
computers and learning the expert system shells. Following the first 4 week introduction, 
students worked directly with their mentors to develop an expert system. 

In acknowledgment of feedback from the 1989 participants, the program was adjusted 
in 1990 and 2 four week schedules were followed, one for twenty students and a second for 
nineteen students. 

The students were introduced to knowledge based system analysis techniques as they 
are applied to engineering, agriculture, and life sciences. This was accomplished through an 
intensive two week class on knowledge engineering. At the end of the first two weeks, the 
students selected a discipline in which to develop an expert system. The discipline focus for 
the second half of the course was in one of the fields of engineering, agriculture, or life 
sciences. The students selected a problem area and worked with a faculty mentor to 
develop an expert system for this area. The system could be a diagnostic tool, an 
identification guide, a management tool, or a system to evaluate an existing simulation 
model. Some examples of the projects that were developed by the students are included in 
Appendix A. 

E. Career Awareness Activities 

A career awareness program was prepared for the students in the program. During 
the first two days, faculty mentors presented their project areas and described career 
opportunities. A variety of fields of science and engineering were presented. The training 
required to pursue various fields was discussed. Scientists and professionals in these fields 
presented their personal experiences and case studies of their work. Industry guests were 
invited to make presentations to the students. During the program, the students took the 
COPS (career orientation placement and evaluation survey) series of tests. The results were 
presented by the career counselor from the College of Engineering. 

F. Philosophy and Ethics of Science Activities 

The students in the program completed an ethics course which was a brief survey of 

3 



6 



various philosophers and the applications of their theories to personal, academic, and 
professional situations. The philosophies of Kant, Socrates, Plato, Aristotle, Mills, and 
Hobbs were presented. The course first introduced the general topic of ethics, and then 
concentrated on the specifics of scientific and engineering ethics. The class discussions 
included examples for scholastic, personal and career situations, and a code of ethics that 
encompassed both private and professional life was developed by each of the students. 

G. Follow-up Activities 

After the first summer program, the project director met with each student at his/her 
high school and described the program in a science class at each school. The students 
presented their expert system to their local class. The students were also encouraged to 
enter their projects in the junior academy of science program, the science and engineering 
fair, and/or the junior science and humanities program. During the second year, the 
Knowledge Based System Development Laboratory was made available to the students for 
further development of their expert system during six weekend return visits to the campus. 
Students were expected to return for at least two of the weekend sessions. 

H. Project Setting 

The Louisiana State University Baton Rouge campus was the site for the program. 
The students were housed in LSU dormitories and the university food service system was 
used for student meals. The LSU Middleton Library on campus was available to the 
students for literature searches. 

The students in this program worked in the LSU Knowledge Based System 
Development Laboratory for Agricultural Sciences. The laboratory was funded through the 
academic enhancement component of the Louisiana Education Quality Support Fund 
(LEQSF). The laboratory has 10 state-of-the-art artificial intelligence work stations (Sun 
Microsystems Sparc Stations). During the first two weeks, the students used the lab to learn 
two expert system shells, 1st Class and EXSYS. 

Two weekend field trips were scheduled, first to visit the Louisiana University Marine 
Consortium camp to study coastal marshes, and a second weekend to visit the NASA facility 
at Bay St. Louis and the Naval Air Museum at Pensacola. Students were also taken on field 
trips to the Ben Hur Aquaculture Center, the T. E. Patrick Dairy Improvement Center and 
the St. Gabriel Research Station, to observe research projects in agricultural sciences. 

I. Program Staff 

The project directors were a professor of dairy science, and a professor of agricultural 
engineering at L^U. One director was responsible for the career awareness and research 
methodology courses and coordinated the student follow-up. The other director was 
responsible for the coordination of the program, the selection of the student participants, 



and was the primary teacher in the knowledge engineering laboratory. Undergraduate 
student counselors worked with the students and coordinated their weekend activities. 
During the first year, an engineering professor taught the expert systems classes. A graduate 
student was hired to provide lab assistance to the students as they developed their expert 
systems. In addition, numerous faculty in the LSU Colleges of Agriculture and Engineering 
worked with the students and presented the courses on career awareness and research 
methodology. Over 30 faculty were involved in the development of different expert systems. 

J. Cost Considerations 

(Note: The evaluator does not have permission to publish detailed expenditure 
information at this time.) 

Total funding from NSF was $177,276. The categories of expenditures included 
salaries, travel, operating services, supplies, professional services, other charges, and indirect 
(overhead) costs. It is estimated that the program required twice as much of the project 
director's effort as was actually charged, and it was noted that the faculty mentors' time (30 
faculty mentors @ $1,000 each) was also not charged to NSF. With fringe benefits and 
indirect costs, this would have added another $105,352 to the cost, for a- total of $290,105 
for the two year program. NSF paid only about 61% of the actual cost for this program. 

The per student cost to NSF for this program was approximately $3,000. Students 
were paid a stipend of $100 per week each for their participation. Subtracting this from the 
NSF costs means the average program cost to NSF was less than $2,500 per participant. 

The project would not have been possible without the KBSDL, which provided the 
computer equipment necessary for developing expert systems and using knowledge 
engineering concepts to model decision making processes. The equipment in this laboratory 
cost $234,000 and it was provided for use in this program at no charge to NSF. 
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III. The Evaluation Study 



A. Justification 

The LSU program director requested evaluation of the Knowledge Engineering for 
Young Scholars Program 1) to document program implementation and operation and to 
provide feedback for the sponsoring agency (NSF); and 2) to assess program effectiveness 
for the LSU program managers as a planning aid for future program proposals. 

B. Study Activities 

This program evaluation was initiated on June 6, 1991 as a graduate level program 
evaluation assignment for Dr. Charles Teddlie's EDAF 7290 class at LSU. Several meetings 
and discussions were held with the program director during June and July, 1991. Documents 
and records pertinent to the program were obtained and reviewed during this time. 
Appointments were made and interviews with the two program managers, faculty mentors, 
teachers, and counselors were conducted on July 16, 18, 19, 22, and 24. Telephone 
interviews with participants, parents of participants, and non-participants were conducted 
August 29 through Sept. 5, 1991. Additional meetings were held with the program director 
in September, while the report was being completed. 

1. For this program evaluation, the following records and documents were reviewed: 

- National Science Foundation Young Scholars Program announcements, press releases, 

correspondence, instruction booklets, brochures, and materials maintained by the LSU 
KEYS program manager. 

- LSU proposals and reports to NSF, correspondence, brochures, and KEYS program 

announcements. 

- Participant and non-participant applications for the LSU KEYS program. 

- Pre- and post-test NSF participant questionnaires. 

2. For this program evaluation, the following people were interviewed for narrative 
information: 

- LSU KEYS program managers (2). 

- KEYS faculty mentors (2). 

- KEYS program teachers (2). 

- KEYS program counselors (2). 

- KEYS program student participants (random sample of 1C). 

- Parents of KEYS student participants (random sample of 6). 

- Applicants to the KEYS program who did not participate (random sample of 10). 
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C. Evaluation Description 



This evaluation is a summative report/outcome assessment design used both to 
document program implementation and completion, and to determine if the LSU 
performance of the KEYS program was consistent with the intent of the Young Scholars 
Program as it was authorized by the National Science Foundation. 

For their own national program evaluation purposes, NSF required that the local 
program managers have participants complete both pre- and post-participation NSF 
Participant Survey Forms. NSF provided pre-participation "Applicant Forms," which 
requested demographic data for each participant, and post-participation "Participant Forms," 
which asked participants to rate the program. Five of the questions included in the pre- 
participation form were repeated in the post-participation form. These five questions were: 

- How interested are you in taking more science and mathematics courses at school? 

- How likely is it that you will become a scientist, engineer or mathematician in the future? 

- As things stand now, how far in school would you like to go? 

- If you plan to go to college, list your (first, second, third) choice of majors. 

- How certain are you of your first choice of major above? 

For this evaluation, KEYS participants' and non-participants' responses to these 
questions were considered. See Section IV. Results, page 17 for a comparison of the 
participants* pre- and post-test responses, and page 14 for a summary of the non- 
participants' responses. 

D. Audience 

The audience for this evaluation includes the LSU and the NSF program managers. 
In addition, and at the LSU program managers* discretion, the evaluation report can be 
made available to other LSU administrators and KEYS program personnel. 

E. Rhetorical and Evaluable Models 

What follows is a summarized evaluability assessment for the NSF Knowledge 
Engineering for Young Scholars program conducted at LSU. 
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NSF Young Scholars Program 
LSU Knowledge Engineering for Young Scholars 

RHETORICAL MODEL EVALUABLE MODEL 



Process goals : 

1. Students will develop expert systems Yes, review LSU reports to NSF. 

in science or engineering. Interview faculty mentors, program 

managers, teachers, students and 
parents. Verify student work. 



2. Students will work with LSU 

faculty mentors and contribute 
to LSU faculty research efforts. 

3. Students will learn about career 

options in science and engineering 
and take the COPS test. 



Yes, review NSF post-test survey 
responses. Interview faculty and 
students. 

Yes, review NSF post-test survey 
responses. Interview teachers, 
students, faculty & program managers. 



Product goals : 

1. NSF - Increase 8- 12th grade students' 

students' exposure to science and 
stimulate students' interest in science. 

LSU - Introduce students to the concepts 
of artificial intelligence, expert and 
knowledge based systems. 

2. NSF - Acquaint students with university 

research. 

LSU - Introduce high ability high school 
students to research @ LSU. 

3. NSF - Build students' confidence to pursue 

careers in science or engineering. 

LSU - Introduce high ability high school 
students to career options in science 
and engineering. 



Yes, review NSF pre- and post-test 
participant surveys. Conduct 
interviews. 

Yes, review LSU reports to NSF 
regarding courses taught. Interview 
program managers, teachers & 
students. 

Yes, review records of the research 
projects the students conducted with 
LSU faculty. 

Interview faculty and students. 

Yes, review NSF post-test participant 
surveys. Interview students, parents, 
teachers, faculty, counselors and 
program managers. 
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IV. Results 



The following questions were used to conduct interviews for this program evaluation. 
A brief summary of some of the comments made in these interviews follows each question. 

A. Interviews with KEYS Program Staff 

Question : What activities were used to achieve program objectives? 

The activities that were seen as contributing to program objectives included class 
instruction, hands-on computer laboratory experience, interaction with faculty mentors, the 
students' oral presentations of their own work, field trips and xours, literature review, 
residential campus life experience, social interaction with other gifted students, teamwork 
on projects, development of their own personal and professional code of ethics, responsibility 
and accountability for their own work, and the rewards of doing independent research for 
personal satisfaction instead of for a grade. 

Question : How did activities lead to objectives? 

The students were introduced to new career opportunities. They also found out they 
had to build support groups and use teamwork to get their projects completed. 

Question : What variables, not independent of the program, facilitated or impeded 
goal accomplishment? 

Program factors which were seen as facilitating goal accomplishment included working 
with mentors, the state-of-the-art computer laboratory, the ethics class, working together with 
buddies or in teams, the changes made to the program in year 2, the fact that the mentors 
were more experienced in year 2, the amount of time the mentor could spend working with 
the student (a few large blocks of time were seen as more productive than many small 
blocks), living in dormitories, structured time, deadlines, a regular reporting schedule, and 
the experience of the directors and mentors in scientific fields. 

Program factors which were seen as impeding goal accomplishment included living 
in dorms, lack of time on the part of some busy faculty mentors, boring history of AI classes 
in year 1, the dorm counselors in year 1, and some other facilities (besides the computer lab) 
were not available for use in the program. 

Question : What did you perceive as the most serious difficulty the program faced in 
meeting its objectives? 

The program director reported that in year one, the difficulty was dormitory 
counselors who had no understanding of what the program participants were doing. They 
were just evening chaperons and didn't understand what the students did during the day in 
the program. These counselors couldn't work with the students in the evenings and just 
functioned in a police role. They also didn't see themselves as role models for the student 
participants. This situation was corrected in year two and the dorm counselors actually 
participated in the program and were role models for the student participants. Also in year 
one, the instructor for the expert systems program course was not working with the students 
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and their mentors to actually develop the systems. This didn't work. In year two, the 
instructors worked with the students and mentors to develop the expert systems. After the 
program, it was difficult to get the students to return to the computer lab during the school 
year to refine their expert systems. They weren't being paid for the follow-up activities and 
the school visits and weekend workshops weren't well attended and didn't seem to 
accomplish much. In fact, follow-up was repeatedly mentioned by the directors as the 
biggest problem. Several approaches were tried and none seemed to be effective. Getting 
mentors was mentioned as a difficulty. Faculty mentors are busy professors and they had 
limited time available. Faculty had to be recruited to participate in the program. In year 
one, the faculty were not experienced in how to proceed, but they did better in year two. 

Question : What do you consider to be the most important thing accomplished in the 
program? 

Through their work in the computer lab, students learned about goals and found that 
there are several ways to accomplish tasks. They learned the importance of teamwork, and 
gained confidence in their own ability to participate in college life. 

Question : What were the overall progran. strengths/weaknesses? 

Strengths - Mentioned most often were the faculty/student mentor r^ationships, the 
LSU facilities, the tours and field trips, and the sophisticated computer lab. The enthusiasm 
of the staff was also seen as a big contribution to the success of the program. 

Weaknesses - As previously noted, the dorm counselors in year one, and the 
ineffectiveness of follow-up efforts during the school year. 

Question : Anything else you wish to add? 

A program director said he wanted the students to have fun, enjoy the experience and 
their interaction with the other students. The other director said that it would be great if 
20% of such talented students would come to college at LSU. The participants learned 
about themselves and others, improved their attitudes regarding science, and increased their 
knowledge about the science of agriculture. A teacher said KEYS had also addressed a 
societal problem. Xids aren't being taught, aren't prepared, and aren't motivated about 
independence, accountability, and personal responsibility. This program introduced them 
to life skills they aren't being taught elsewhere. 

B. Interviews with Parents of KEYS Participants 

Question : What do you think your child got out of participating in the KEYS 
program? 

Responses included exposure to college life; in-depth work, comfort and confidence 
with computers; personal growth, self-confidence, and enhanced self-esteem from the 
interaction with faculty mentors; heightened enthusiasm for a career in science; and the 
opportunity to see how much expertise there is at LSU. 

Question : Were you satisfied with the material presented to your child? 
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Responses were all positive, and comments included a desire for the program to be 
offered again with a wider range of science offered for the projects* The material turned 
the students on to computers, and the use of tours and field trips to see applications of 
science really impressed the students with what is possible for them career-wise. 

Question : Would you allow another child to attend a similar program? Would you 

be willing to pay? How much? 
The responses were all yes to parts one and two, and part three responses ranged 
from $200 to $500, with $200 being the most frequent response (3 of 6). 

Question : Anything else you remember, or wish to say? 

The LSU people were very nurturing and KEYS was a good orientation to college 
life for my son. Please tell the project director it was a wonderful growth experience, a real 
gift* Over all, it was a well run program* KEYS reinforced my daughter's science interests. 
LSU is losing out by not recruiting and paying these G&T kids to attend LSU. There has 
been little or no LSU recruiting of my daughter. My daughter loved the KEYS program. 
Being paid made it possible for her to attend. Without the pay, she would have had to get 
a summer job. There was good parent involvement. The field trips were a very good 
experience for my son. It was an excellent program. 

C. Interviews with KEYS Participants 

(Initial greeting conversation was used to determine how they were doing personally, 
if they had selected a college and a major for college, how they had done/were doing in high 
school, what math and/or science courses they had taken since participating in the KEYS 
program, and after the first few interviews, an assurance that there were no "right or wrong" 
answers for the questions.) 

Question : How has your participation in the KEYS program influenced you in 
selecting courses to take in high school, selecting a college to attend, 
or selecting a major in college? 

Female high school senior: In high school, I want to take more computer courses and 
learn more about computers* I don't know what college I want to attend yet. I plan to get 
an MD or PhD and conduct clinical research. 

Male high school senior: I'm taking computer science now, also physics, science and 
math, and Fm making A's, I'm considering LSU and haven't decided yet. I'm also 
considering a computer science major* 

Female high school senior: I'm taking science and math courses in school and doing 
good* I haven't picked a college yet, but I'll major in engineering. 

Female high school student: Definitely influenced what courses I'm taking in high 
school. I'm in physics this year and I took algebra 2 and chemistry last year. I'm undecided 
about which college, but think I'll major in advertising. 

Male high school senior: KEYS broadened my mind regarding sciences. Fm more 
interested now. I've taken chemistry, physics, algebra 2, and trig, and'I have a 4.5 GPA with 
honors- Duke is my first choice, and a political science major. 
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Female high school student: Fm taking more science courses now because of KEYS. 
Fm 19th in my class with a 4.1 GPA* I'm taking physics and trig, and engineering 
technology. I'm going to Southeastern to major in nursing. 

Female college freshman: Yes, I took physics and calculus after the KEYS program. 
Now Fm at LSU majoring in chemical engineering. 

Question : Has your KEYS participation modified the way you make personal 
decisions? 

(After repeatedly getting "no" answers, this question was modified to: "Has your 
participation in the KEYS program, learning about artificial intelligence, and developing 
your own expert system influenced the way you make decisions?") 

Fve matured about decisions regarding what courses to take. Fm more logical now. 
Fm weighing decisions, especially those regarding what classes to take. 

Question : What portions of the program have had the greatest impact on you? 

The field trips, getting to see what people do in science careers, friendships with the 
other students, the college experience, the expert systems, working with computers, and the 
interactions with the professors were a real confidence booster. 

Question : What was the major strength of the KEYS program? 

The people who ran the program. KEYS was well organized. We got to work on 
our own. The KEYS set-up with lectures, computers, field trips and the college environment 
worked for me. Working as a group and team effort, I found out that trying to do it all on 
my own didn't work. KEYS was well structured. The weekend field trips were good and 
KEYS wasn't boring. We got to do real research and work with computers. 

Question : What was the major weakness of the KEYS program? 

Some of the class presentations were boring because the material was dry. 

Question : Would you have attended the KEYS program if you weren't paid? If not, 
why? If yes, how much would you have paid to participate in the 
program? 

All responded positively to part one, and part two responses ranged from $75 to $200, 
with $200 being the most common response (4 of 6). 

Question : Any ^ling else you remember, or wish to say? 

It was the best summer program Fve been to. Lots of fun. I loved it. It was an 
experience you never forget. I made friends for the rest of my life. The field trips were very 
interesting. KEYS was a great program. 

D. NSF Post-participation Evaluation 

The eight questions NSF included in the post-participation form for students to use 
to rate the program were: 
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(This is the scale which was 4 = strongly agree 
used to rate these questions.) 3 = agree 

2 = disagree 

1 = strongly disagree 

_ 1. Participation in this project has increased my interest in science. 

_ 2. Participation in this project has increased my understanding of the research process. 

_ 3. The financial aid offered me, if any, had a strong influence on my decision to 

participate in this project. 
_ 4. I plan to take more math and/or science courses in the future than I had originally 

planned as of the end of the school year. 
_ 5. For the first time I now understand the course requirements for majoring in science. 
_ 6. I will recommend projects like this to friends who are interested in science or 

mathematics. 

7* If I had the opportunity, I would participate in a similar project. 

_ 8. I spent a lot of time with the scientists/mathematicians/engineers in this project. 

In addition, a comments section was provided on the post-participation form. 
(Participant comments were very positive about the program.) 



LSU Participants' Post-participation Survey Form Program Ratings 
(Mean Scores*) 
Session 1 Session 2 Session 3 

(Question) 



L 3.6 3.32 3.58 

2. 3.65 3.32 3.74 

3. 3.2 3.37 3.32 

4. 2.7 2.53 2.95 

5. 2.65 2.58 2.74 

6. 3.8 4.0 3.95 

7. 3.95 3.84 3.79 

8. 3.5 3.63 3.47 



(*The 4 point scale used to rate these questions is noted above.) 

As can be seen from these results, only two questions consistently received ratings 
which averaged less than 3 (agree) by the participants in all three program sessions: 

#4, I. plan to take more math and/or science courses in the future than I had originally 

planned as of the end of the school year. 
#5, For the first time I now understand the course requirements for majoring in science. 

For comments, the students noted that they were already taking as many math and 
science courses as they could, and therefore they couldn't plan to take more; and also, that 
they already knew the course requirements for majoring in science and this program was not 
the first time this information had been presented to them. 
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Otherwise, as can be seen from the above ratings, the students agreed or strongly 
agreed that the LSU KEYS program increased their interest in science; increased their 
understanding of the research process; the financial aid offered to them had a strong 
influence on their decision to participate in the project; they would recommend projects like 
this to friends who are interested in science or mathematics; if they had the opportunity, they 
would participate in a similar project; and they spent a lot of time with the scientists and 
engineers in this project, 

E. Telephone Survey of KEYS Participants 

Ten KEYS program student participants were surveyed by telephone the first week 
of Sept., 1991, to determine their most recent choice of college ajid major. The results were 
as follows: 



Current status 


College choice 


Major 


1. HS senior 


undecided* 


MD or PhD, clinical research 


2. Univ., Yr.l 


LSU 


biological engineering 


3. HS senior 


undecided* 


computer science 


4. Univ., Yr.l 


Spelman 


chemical engineering 


5. HS senior 


undecided 


aerospace engineering 


6. HS senior 


undecided 


advertising 


7. HS senior 


Duke 


political science 


8. HS senior 


Southeastern 


nursing 


9. HS senior 


N. Carolina* 


science/pre-med. 


10.Univ., Yr.l 


LSU 


chemical engineering 



♦These students reported that they were also considering LSU. 



F. Telephone Survey of Non-Participants 

The following questions were used to interview students who applied for the KEYS 
program but did not participate in KEYS. Ten students were surveyed by telephone the first 
week of Sept., 1991, and their responses are noted. Non-participant student survey 
questions: 

1. How interested are you in taking more science and mathematics courses at school? 

5 extremely interested 

5 very interested 

somewhat interested 

not very interested 

2. How likely is it that you will become a scientist, engineer or mathematician in the future? 

JL a very good chance (better than 50%) 
J, a even chance (50%) 
1 not a very good chance (less than 50%) 
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3. As things stand now, how far in school would you like to go? 

won't finish high school 

would like to graduate from HS, but won't go any farther 

would like to &o to vocational, trade or business school after HS 

would like to attend college 

2 would like to graduate from college 

8 would like to attend higher level of school after graduating from college 

4. If you plan to go to college, what is your choice of majors? 

first choice * 

second choice 

third choice 



5. How certain are you of your first choice of major? 

6 very certain 
3 fairly certain 
1 not certain 

6. Do you know where you are going to college? 

Responses: Harvard (1), N. Carolina State (1), undecided (1), no (3), LSU (3), 
McNeese State (1) 

* First choice responses given for #4: biochemistry (2), electrical engineering (1), 
undecided/journalism (1), political science (1), zoology (1), engineering (1), chemical 
engineering (2), biological systems engineering (1) 

G. Participant & Non-participant College Major Choices 

As NSF's program goal was to recruit more students into careers in science and 
engineering, the KEYS participants' college major choices were compared to non- 
participants' college major choices to determine what effect the KEYS program had on 
student participants. As can be seen from the results shown below, this comparison of 
KEYS participants to a control group of non-participants showed no difference between the 
two groups: 



Sept.. 1991 Telephone Survey 



Science/Math/Engineering Major 
chosen not chosen 



total 



KEYS sample 



8 



2 



(10) 



No KEYS sample 



8 



2 



(10) 



total 



(16) 



(20) 



15 
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H. Participant's Pre- and Post-participation NSF Survey Choice of College Major 

Analysis of the KEYS participants' pre-and post-participation questionnaire responses 
regarding their choice of science, math or engineering as a first choice of major also showed 
a result of no significant difference. A Chi Square test, the McNear test for significance of 
change, was used to compare nominal data for a two-sample case with dependent samples. 
This test is used in pre-test/post-test designs in which the same sample of subjects is 
categorized before and after some intervening treatment. The null Hypothesis is that there 
will be no difference between cells A and D. An equal number of changes are expected in 
both directions (the number of yes to no changes will equal the number of no to yes 
changes), and the frequency in cell A will equal the frequency in cell D. The null was not 
rejected. 

Science, Math or Engineering as 1st Choice of Major: 



19 89 Program 



After 

KEYS 

Program 

total 



NO 
YES 



Before Keys Program 
YES NO 



1 (A) 
18(C) 
19 



0 (B) 

1 (D) 
1 



total 
1 

19 
20 



After 

KEYS 

Program 



Before Keys Program 



1990 ProEram. Session #1 

YES 

NO 0(A) 
YES 14 (C) 

total 14 
*One of the 20 participants did not complete the questionnaire. 



NO 
4 (B) 
1 (D) 
5 



1990 Program. Session #2 



Before Keys Program 
YES " NO 



After 

KEYS 

Program 

total 



NO 
YES 



1(A) 
13 (C) 
14 



5 (B) 
0 (D) 
5 



total 
4 
15 
19* 



total 
6 
13 
19 
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I. Comparison of Participant's Pre- and Post-participation Responses 



In comparing the five questions which were repeated on the NSF pre- and post-test 
surveys, only one response difference was noted. As the KEYS participants were all high 
ability students to start with, it is not surprising that 100% had noted on the pre- and post- 
test that they would like to at least attend college. However, there was an overall post-test 
increase of 1% in the number of students who said they would like to attend a higher level 
of school after graduating from college. The increase came from among those students who 
had said on the pre-test that they would like to (just) graduate from college. After the 
KEYS program, these students reported that they were now interested in attending more 
school after college. 



V. Comments and Conclusions 

During the evaluation, it became apparent to this evaluator that the survey instrument 
data would not be adequate for assessment of goal attainment as the students were all high 
ability individuals, most of whom had entered the program with a stated commitment or 
strong interest in a career in science, engineering or mathematics. It was not surprising that 
there was little or no difference between pre- and post-test responses for most of the 
participants (and non-participants). The NSF instruments did not document the enhanced 
confidence and comfort levels with computers and teamwork which were reported in the 
interviews. 

Additionally, it was noted that the student participants' self report on their confidence 
level for their first choice of a major in college often did not correspond with their 
subsequent actions. For example, one student said he was only "fairly" confident of his first 
choice of major (biology). This student is currently enrolled at LSU in biological 
engineering, has several scholarships, and reports a high level of satisfaction with his choice. 
Another student said she was "not" confident of her first choice of major (engineering). This 
student is currently enrolled at Spelman in chemistry, plans to be a chemical engineer, also 
has several scholarships, and her father reports that she, too, has a high level of satisfaction 
with her choice. In other words, the NSF post-test survey responses of the students proved 
to be somewhat unreliable when compared to the students' subsequent actions. 

Since the program was very financially attractive to students (they were paid $100 per 
week and provided room and board to attend), and since the announcements for the 
program noted that applicants were required to describe their interests and achievements 
in science and math, there was the possibility that some students "cheated" when they 
applied for the program and completed the NSF pre-test. These students may not have 
been completely honest when they declared they were very confident of their first choice of 
a science major on the pre-test. This would certainly explain some of the students' response 
switches from very confident about a science major on the pre-test to not confident on the 
post-test, or a very confident response with a first choice switch to a non-science major on 
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the post-test. For example, one student stated on the pre-test that he was "fairly" confident 
of a first choice of engineering as his major. On the post-test, he said he was "very" 
confident of his first choice of law. Another student who said she was "not" certain of her 
first choice of engineering on the pre-test said she was "fairly" certain of her first choice of 
corporate law on the post-test. Again, both the pre- and post-test student responses on the 
NSF survey proved to be somewhat u: reliable. 

Based on information obtained during telephone interviews with KEYS program 
participants and the parents of participants, the KEYS program participants did in fact work 
with LSU faculty, they developed expert systems, and learned about careers in science and 
engineering. The students reported that they learned more about research and their 
confidence in their own ability to pursue a career in science was enhanced. As these goals 
of the program according to NSF and LSU were accomplished, this evaluation study 
determined that the KEYS program was effective and operated by LSU consistent with the 
intent of the Young Scholars Program as it was authorized by NSF. 

Interviews determined that both parents and students were well pleased with the 
operation and content of the program, and the impact of the program on the students' 
competence and confidence to pursue additional course work in science and mathematics. 
If this program was "the" difference for some of these students in whether or not they gained 
the confidence and teamwork skills they needed to successfully pursue a career in science, 
then the $3,000 price per student to NSF was probably a very good investment. 

VI. Recommendations 

Based on information obtained from comments on the post-participation 
questionnaires and the evaluation interviews, changes made after year one enhanced the 
program in year two. A four week program was preferred by the students over an eight 
week program, therefore it is recommended that future summer programs be kept to four 
weeks or less. Having the counselors and teachers participate in the whole program is more 
effective than having them each do only their own little part. 

Field trips and tours were repeatedly mentioned by parents and students as being 
educational and an effective way to demonstrate science career options. Students repeatedly 
mentioned working in teams and with a buddy as one of the most useful learning 
experiences in the program. Students and parents reported that the use of sophisticated 
state-of-the-art computers, working directly with faculty mentors, the introduction to science 
careers, and having a residential college life experience, were factors which enhanced the 
students' confidence and performance in taking more science, math and computer classes 
in high school and college. 

The program directors assessed post-program follow-up with the students as being 
ineffective. Several different methods were utilized, both going to the students' high schools 
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and inviting the students back to the university. Though NSF requires "follow-up" with the 
participants, it is not clear what NSF's expectations are regarding what this follow-up will 
accomplish. One possible way to continue to make the students feel part of the program 
would be to periodically send out questionnaires with postage-paid return envelopes, which 
the students could use to report on their school and personal activities and successes. This 
information could be used for a quarterly newsletter to update the students on the 
participants' activities, to announce upcoming programs and recruitment visits to their 
schools, to report on the research of their faculty mentors or teachers, to announce special 
opportunities which may be of interest to them, etc. Some parents said they were 
disappointed that LSU hasn't been more active in recruiting these students since the 
program. A program newsletter could be used for this purpose too. And, more recruitment 
coordination between the university recruiters and the program directors is recommended. 

All the parents and students agreed that they would participate in the program if it 
were offered again. Some said they would like to see more areas of science included in 
future program offerings. Recruiting faculty participation for KEYS was seen as difficult, 
therefore, in order to recruit even more science and engineering faculty in the future, it may 
be necessary to get the support of appropriate deans and department heads, perhaps as an 
advisory group for the program. In order to do this, it may be necessary to get the provost 
or chancellor to promote the program to the deans as a gifted student recruitment effort. 
As was noted by parents, staff and students, an enthusiastic staff is imperative. 

Pay for participation was a critical factor for those students who needed summer jobs. 
The $100 per week was an effective inducement. Interestingly, both parents and students 
said they were so pleased with the program that they would be willing to pay for future 
programs like this. $200 was mentioned most often as what they would be willing to pay, 
however, some said as much as $500 to cover room and board. Some version of this 
program could be offered again for a fee, even if it isn't funded again by NSF. Industries 
could be asked to provide scholarships for students who qualified but couldn't afford to pay, 
and the residence halls may wish to contribute room and board to recruit the students to the 
university. 

NSF evaluations of this LSU KEYS program and other NSF Young Scholars 
Programs should not be based solely on data obtained from the pre- and post-test 
participant survey forms. As has been noted here, the information provided on these forms 
was found to be unreliable for evaluation purposes. Interviews with participants and their 
parents yielded more reliable information regarding college, major and career choices and 
the subsequent actions which have been taken by the students. 
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Appendix A. Sample Abstracts of KEYS Students' Expert Systems 



Project Title: Application of Waste Materials to Land as a Soil Amendment 

Mentor: Sam Feagley Department: Agronomy 

Students: (2) 

Brief Abstract: The amount of materials being land filled must be reduced by 25% in 
Louisiana by 1992. Sound decisions based on chemical, physical, mineralogical, and 
biological analyses must be made to determine the material's suitability for land application. 
An expert system was developed to assist in the evaluation of the feasibility of land 
application of paper mill sludge. 

Project Title: Dairy Herd Profitability 

Mentor: Wayne Gauthier Department: Agricultural Economics and Agribusiness 

Students: (4) 

Brief Abstract: Profit is critical to economic survival. A dairyman's ultimate objective is the 
generation of sufficient profits to insure financial independence by some uniquely defined 
time in the future. An expert system was developed as a tool for assessing the profitability 
of a dairy herd. The determinants of profitability and their sets of standard measures for 
building the system were available from the literature. Budget data developed by the 
Department of Agricultural Economics and Agribusiness were used to test the system for 
relevance and reliability. 

Project Title: Embryo Viability 

Mentor: Robert A- Godke Department: Animal Science 

Students: (9) 

Brief Abstract: A data bank on embryos was used to develop a knowledge-based learning 
system. The system assists users in the selection of embryos with a high probability of being 
successfully transplanted. 

Project Title: Aquaculture 

Mentor: Thomas Lawson Department: Agricultural Engineering 

Students: (7) 

Brief Abstract: A general overview of aquaculture in Louisiana was presented. Expert 
systems were developed to assist in selection of aerators, pond size, stocking density and 
profitability of aquaculture systems. 

Project Title: Coastal Erosion 

Mentor: Don Cahopn Department: Coastal Studies 

Students: (2) 

Brief Abstract: Louisiana loses over 50 square miles of wetlands annually to coastal erosion. 
Several techniques have some possibility of reducing the rate of loss. These include: fresh 
water diversions, revegetation with salt tolerant varieties, shoreline stabilization and marsh 
management. This project captured the decision making process of experts in selecting the 
most appropriate environmental management strategy. 
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